Before we can start with our machine learning model we need to understand the relationship between the two variables, therefore we should calculate covariance. This measures the direction of a relationship between the two variables.
library(tidyverse)
library(writexl)
library(plyr)
library(lubridate)
library(plotly)
library(dplyr)
library(corrplot)
library(formatR)
library(htmltools)
library(ggpubr)
library(tufte)
btc_exchange_rate_history <- read.csv("D:/Suli/Szakdolgozat1/development_n_stuff/aggregated_data.csv") %>%
select(-X) %>%
mutate(created_at = as_date(created_at))
btc_usd_tweets_combined <- read.csv("D:/Suli/Szakdolgozat1/clean data/Bitcoin_exchage_rates.csv") %>%
mutate(Date = as_date(Date)) %>%
mutate(year = year(Date), month = month(Date), day = day(Date)) %>%
mutate(Date = make_date(year, month, day)) %>%
mutate(xchange_rate_change = Close - Open) %>%
filter(Date >= "2019-01-01" & Date <= "2022-03-30") %>%
inner_join(btc_exchange_rate_history, by = c(Date = "created_at")) %>%
select(c(-year, -month, -day))
btc_sent_lineplot <- plot_ly(data = btc_usd_tweets_combined, x = ~Date, y = ~xchange_rate_change,
name = "BTC change", type = "scatter", mode = "lines", color = "red") %>%
add_trace(data = btc_usd_tweets_combined, x = ~Date, y = ~daily_avg_sent, yaxis = "y2",
name = "Avg. sentiment", mode = "lines", color = "blue") %>%
layout(title = "Bitcoin exchange rate change compared to previous day's closing rate",
margin = list(t = 150), legend = list(x = 1.1), paper_bgcolor = "rgb(255, 255, 255)",
plot_bgcolor = "rgb(255, 255, 255)", xaxis = list(title = "Date", range = list("2019-01-01 00:00:00",
"2019-12-31 23:59:59"), rangeslider = list(type = "date", visible = T),
list(dtickrange = list(NULL, 1000), value = "%H:%M:%S.%L ms"), list(dtickrange = list(1000,
60000), value = "%H:%M:%S s"), list(dtickrange = list(60000, 3600000),
value = "%H:%M m"), list(dtickrange = list(3600000, 86400000), value = "%H:%M h"),
list(dtickrange = list(86400000, 604800000), value = "%e. %b d"), list(dtickrange = list(604800000,
"M1"), value = "%e. %b w"), list(dtickrange = list("M1", "M12"),
value = "%b '%y M"), list(dtickrange = list("M12", NULL), value = "%Y Y"),
rangeselector = list(buttons = list(list(count = 1, label = "1M", step = "month",
stepmode = "backward"), list(count = 6, label = "6M", step = "month",
stepmode = "backward"), list(count = 1, label = "1Y", step = "year",
stepmode = "backward"), list(count = 1, label = "YTD", step = "year",
stepmode = "todate"), list(step = "all", label = "ALL"))), list(dtick = "M1",
tickformat = "%b\n%Y", ticklabelmode = "period")), yaxis = list(title = "BTC exchange rate change",
range = c(min(btc_usd_tweets_combined$xchange_rate_change), max(btc_usd_tweets_combined$xchange_rate_change)),
gridcolor = "rgb(255,255,255)", showgrid = TRUE, showline = FALSE, showticklabels = TRUE,
tickcolor = "rgb(140, 140, 140)", ticks = "outside", zeroline = FALSE),
yaxis2 = list(title = "Daily average sentiment", overlaying = "y", side = "right",
range = c(min(btc_usd_tweets_combined$daily_avg_sent), max(btc_usd_tweets_combined$daily_avg_sent))))
btc_sent_lineplot
plot(density(btc_exchange_rate_history$daily_avg_sent), type = "n", main = "Distribution of average daily sentiment")
polygon(density(btc_exchange_rate_history$daily_avg_sent), col = "red", border = "gray")
plot(density(btc_usd_tweets_combined$xchange_rate_change), type = "n", main = "Distribution of Bitcoin exchange rate change")
polygon(density(btc_usd_tweets_combined$xchange_rate_change), col = "red", border = "gray")
These seem like pretty standard normal distributions, so we can use Pearson Correlation Coefficient calculation later on.
btc_sent_scatterplot <- plot_ly(data = btc_usd_tweets_combined, y = ~daily_avg_sent,
x = ~xchange_rate_change, marker = list(size = 4, color = "rgba(255, 182, 193, .9)",
line = list(color = "rgba(152, 0, 0, .8)", width = 1))) %>%
layout(yaxis = list(title = "Daily average sentiment"), xaxis = list(title = "BTC exchange rate change (USD)")) %>%
htmltools::div(align = "center")
btc_sent_scatterplot
The plot shows that there is no distinguishable relationship between these variables, but I still wanted to do the covariance and correlation calculation to get the mathematical results.
btc_cov <- cov(btc_usd_tweets_combined$daily_avg_sent, btc_usd_tweets_combined$xchange_rate_change,
method = "pearson")
btc_cov
## [1] 3.301177
A positive covariance means that the two variables tend to increase or decrease together. Correlation helps us analyze the effect of changes made in one variable over the other variable of the dataset. Now that we know this, we should calculate the strength of the relationship between two, numerically measured, continuous variables.
btc_cor <- cor(btc_usd_tweets_combined$daily_avg_sent, btc_usd_tweets_combined$xchange_rate_change,
method = "pearson")
btc_cor
## [1] 0.1162632
One of the most common ways to quantify a relationship between two variables is to use the Pearson correlation coefficient, which is a measure of the linear association between two variables.
It always takes on a value between -1 and 1 where:
- -1 indicates a perfectly negative linear correlation between two variables
- 0 indicates no linear correlation between two variables
- 1 indicates a perfectly positive linear correlation between two variables
Often denoted as r, this number helps us understand the strength of the relationship between two variables. The closer r is to zero, the weaker the relationship between the two variables.
A weak correlation indicates that there is minimal relationship between the variables.
After reading some scientific papers I concluded that continuing down this path would bear no plausible outcome, so I decided to look at some other trend measures that seem promising.
Downloaded the data for Google search popularity of Bitcoin through Trendecon and Google Trends API. Because of the limitations of standard Google Trends data I had to use the Trendecon package’s rts_gtrends_mwd function to build a consistent daily time series.
Construct a robust and consistent daily Time Series from Google Trends data. Daily, weekly and monthly Data is downloaded and consistently aggregated, using the Chow-Lin methodology.
btc_ggl_trends <- read.csv("D:/Suli/Szakdolgozat1/data_to_be_cleaned/btc_google_trends.csv")
head(btc_ggl_trends)
## X time value
## 1 1 2019-01-01 1.054060
## 2 2 2019-01-02 17.290539
## 3 3 2019-01-03 19.200650
## 4 4 2019-01-04 15.380214
## 5 5 2019-01-05 7.739413
## 6 6 2019-01-06 7.739342
Checking distribution:
plot(density(btc_ggl_trends$value), type = "n", main = "Distribution of Google Trends data")
polygon(density(btc_ggl_trends$value), col = "red", border = "gray")
btc_usd_ggltrnds_combined <- read.csv("D:/Suli/Szakdolgozat1/clean data/Bitcoin_exchage_rates.csv") %>%
inner_join(btc_ggl_trends, by = c(Date = "time")) %>%
mutate(xchange_rate_change = Close - Open) %>%
mutate(value = ifelse(xchange_rate_change < 0, value * -1, value)) %>%
select(-X) %>%
dplyr::rename(Trends_indicator = value)
btc_ggltrnds_scatterplot <- plot_ly(data = btc_usd_ggltrnds_combined, y = ~Trends_indicator,
x = ~xchange_rate_change, marker = list(size = 4, color = "rgba(255, 182, 193, .9)",
line = list(color = "rgba(152, 0, 0, .8)", width = 1))) %>%
layout(yaxis = list(title = "Google Trends indicator"), xaxis = list(title = "BTC exchange rate change (USD)")) %>%
htmltools::div(align = "center")
btc_ggltrnds_scatterplot
Plotting the absolute values shows the linearity between Google Trends indicator and BTC exchange rate change:
ggl_trends_abs_scatterplt <- plot_ly(data = btc_usd_ggltrnds_combined, y = ~abs(Trends_indicator),
x = ~abs(xchange_rate_change), marker = list(size = 4, color = "rgba(255, 182, 193, .9)",
line = list(color = "rgba(152, 0, 0, .8)", width = 1))) %>%
layout(yaxis = list(title = "Google Trends indicator"), xaxis = list(title = "BTC exchange rate change (USD)")) %>%
htmltools::div(align = "center")
ggl_trends_abs_scatterplt
ggscatter(btc_usd_ggltrnds_combined, x = "xchange_rate_change", y = "Trends_indicator",
add = "reg.line", conf.int = TRUE, cor.coef = TRUE, cor.method = "pearson", xlab = "BTC exchange rate change (USD)",
ylab = "Google Trends indicator")
When you perform a statistical test a p-value helps you determine the significance of your results in relation to the null hypothesis. A p-value less than 0.05 (typically ≤ 0.05) is statistically significant. It indicates strong evidence against the null hypothesis, as there is less than a 5% probability the null is correct (and the results are random). Therefore, we reject the null hypothesis, and accept the alternative hypothesis. The alternative hypothesis states that the independent variable did affect the dependent variable, and the results are significant in terms of supporting the theory being investigated (i.e. not due to chance).
Covariance:
btc_ggl_cov <- cov(btc_usd_ggltrnds_combined$Trends_indicator, btc_usd_ggltrnds_combined$xchange_rate_change,
method = "pearson")
btc_ggl_cov
## [1] 31092.77
Correlation:
btc_ggl_cor <- cor(btc_usd_ggltrnds_combined$Trends_indicator, btc_usd_ggltrnds_combined$xchange_rate_change,
method = "pearson")
btc_ggl_cor
## [1] 0.7332396
There is a moderate uphill (positive) relationship.
Chosen to improve model accuracy: